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Large-scale scale testing agencies, which are generally located within provincial 
ministries of education and the Council of Ministers of Education, Canada, have 
called for secondary data analysis of test and survey data that link contextual and 
educational factors to student performance. First, the potential of secondary analyses 
is outlined, followed by discussion of seven pitfalls that need to be addressed to 
realize this potential in Canada. Greater collaboration between testing agencies and 
teams of qualified secondary analysis researchers needs to take place to ensure that 
policy issues relevant to education are addressed in meaningful ways. 
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Les organismes responsables des epreuves communes, qui relevent gmeralement des 
ministeres provinciaux de I'Education et du Conseil des ministres de I’Education 
(Canada), ont demande une analyse de deuxieme ordre des donnees d' epreuves et 
d'enquetes reliant des facteurs contextuels et pedagogiques au rendement des eleves. 
Le potentiel des analyses de deuxieme ordre est d'abord presente ; suit une 
discussion des sept ecueils a eviter pour en tirer profit au Canada. Une meilleure 
collaboration entre les organismes responsables des tests et les equipes de chercheurs 
specialises dans les analyses de deuxieme ordre doit etre mise en place afin d'assurer 
que les questions de politiques en matiere d'education sont abordees de maniere 
pertinente. 
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Large-scale testing agencies, which in Canada are generally located 
within provincial ministries of education and the Council of Minisfers of 
Educafion, have called for secondary dafa analysis of fesf and survey 
dafa fhaf link confexfual and educational facfors fo sfudenf performance. 
Tesfing agencies complefe fhe firsf or primary analysis of fhe dafa 
collecfed. These analyses generally consisf of summary sfafisfics 
aggregafed af sfudenf, school, disfricf, and/or provincial levels, and are 
complefed eifher by personnel wifhin fhe agencies or by exfernal 
confracfors. However, agency officials believe fhaf fhe dafa can be 
analyzed in differenf ways and fhaf deeper, more comprehensive 
analyses fhaf relafe variables af fhe sfudenf, class, and school levels fo 
sfudenf achievemenf measures can be conducfed. Consequenfly, fhey 
make dafa available fo independenf researchers fo perform secondary 
dafa analyses designed fo beffer answer fhe original questions or fo 
answer new questions. Such secondary analyses increase fhe cosf 
effecfiveness of fesfing programs because dafa fhen serve a broader 
specfrum of research quesfions and researchers. The primary research 
quesfions are addressed and also a series of secondary, more complex 
and deeper quesfions. Answers fo fhese quesfions are pofenfially of 
greafer inferesf fo adminisfrafors and officers of fesfing agencies and fhe 
clienfs fhey serve. Typically, secondary dafa analysfs, who are locafed af 
universifies, are independenfly funded, fhereby adding an aura of 
aufonomy and credibilify fo fhe secondary resulfs and findings. 

If is possible fo see secondary dafa analyses of provincial fesfing dafa 
(e.g., Brifish Columbia, Alberfa, and Onfario), national fesfing dafa (fhe 
liferacy, mafhemafics, writing, and science assessmenfs conducfed by fhe 
Council of Minisfers of Educafion), and infernafional fesfing dafa (e.g.. 
Third Infernafional Mafhemafics and Science Sfudies [TIMMS] 
conducfed by fhe Infernafional Educafion Association and fhe 
Programme for Infernafional Sfudenf Assessmenf [PISA] conducfed by 
fhe Organization of Economic and Cooperafive Developmenf). Sfafisfics 
Canada encourages secondary dafa analyses fhrough fhe provision of 
dafa sefs in Research Dafa Cenfres locafed af universifies across fhe 
counfry, and offers aid and awards fo graduafe sfudenfs who perform 
secondary analyses using Sfafisfics Canada dafa sefs (e.g., Nafional 
Longifudinal Sfudy of Children and Youfh [NLSCY]). Financial supporf 
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for such secondary data analysis studies can also come from the Social 
Sciences and Humanities Research Council in the form of support 
through the investigator-framed research program (Social Sciences and 
Humanities Research Council of Canada, 2005) 

Experiences with secondary data analyses in Canada have been 
mixed. Secondary data analysts have found problems with the data they 
receive, including but not limited to (a) sampling designs that restrict 
detailed analysis, (b) the failure to collect data on relevant variables 
needed for statistical modeling, (c) lack of linkages between contextual 
and achievement data within and between the levels of analysis, (d) 
missing data due to non-response, (e) inaccurate student responses to 
both the survey and achievement items, and (f) incomplete or incorrect 
documentation.! Consequently questions are raised about the integrity of 
the data and the subsequent validity and usefulness of the results and 
findings. 

OUR POSITION 

The authors of this article have completed a series of large-scale 
secondary analyses of the 2001 Mathematics and 2003 Writing 
assessments, which are part of the School Assessment and Indicators 
Project (SAIP) of the Council of Ministers of Education, Canada (CMEC). 
The primary intents of these secondary analyses were to develop and 
validate empirical models that relate indicators of educational resources 
and conditions, home and school traits, to student achievement on 
significant learning outcomes within a policy-relevant framework. These 
models can enhance understanding of school performance, particularly if 
focused on variables that have implications for educational policy and 
classroom practice. A collateral outcome is the identification of the data 
collection strategies for improving the quality and utility of future large- 
scale assessments. 

Our experience has suggested that our initial reservations and 
cautions about secondary data analyses were well founded. However, 
we are convinced that the basic concept of secondary data analyses of 
SAIP, and other large-scale data sets, has merit and can contribute to 
education in Canada. Further, we believe seven issues are central to the 
success of secondary data analyses. The first three are concerned with 
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the nature of secondary data analysis outcomes, the need for clearly 
esfablished links befween fhe primary agency responsible for fhe 
assessmenf and secondary dafa analysis researchers, and fhe need for 
mulfidisciplinary secondary dafa analysis research feams wifh adequafe 
infrasfrucfure supporf. The remaining four issues deal wifh fechnical 
quesfions fhaf offen arise abouf fhe nafure and qualify of fhe dafa for 
secondary dafa analyses. 

1. What is the nature of the results from the secondary data analyses of large- 

scale data sets in education ? 

If is imporfanf fo recognize fhe nafure of secondary dafa analyses and 
whaf fhey can and carmof do. Sfudies, such as fhose available from 
CMEC, can be besf described as explanafory, nonexperimenfal research 
(Johnson, 2001). Kerlinger (1986), for example, provided fhe following 
definition of nonexperimenfal research: 

Nonexperimental research is systematic empirical inquiry in which the scientist does not 
have direct control of independent variables because their manifestations have already 
occurred or because they are inherently not manipulable. Inferences about relations 
among variables are made, without direct intervention, from concomitant variation of 
independent and dependent variables, (p. 348; italics in original) 

Thus, secondary analyses of dafa are nonexperimenfal in fhaf fhe 
dafa have already been collecfed from nafurally occurring groups of 
sfudenfs in classes and schools. Consequenfly, if is nof possible fo 
manipulafe fhe condifions and confexf, even if fhese variables could be 
manipulafed in pracfice. Therefore, fhere is no sense of cause-and-effect as 
found in experimenfal research (Cook & Campbell, 1979). Insfead, 
secondary dafa analyses, which identify associafional relationships, are 
explanafory because fhe analyses can identify only explanatory factors fhaf 
help explain variation in fhe dependenf variable, which, in many cases, 
is sfudenf achievemenf. As Pascarella and Terenzini (2005, p. 636) 
poinfed ouf, fhe besf approach for fhis purpose is causal modeling (e.g., 
sfrucfural equafion modeling [Byrne, 1994; Joreskog & Sorbom, 1993]) 
and hierarchical linear modeling (Raudenbush & Byrk, 2002). However, 
fhe inferprefafion of fhe resulfs yielded by fhese models musf be made 
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carefully to avoid claiming a cause-and-effect relationship between the 
independent variables and student achievement. Follow-up 
experimental and quasi-experimental studies need to be undertaken to 
confirm such tentative cause-and-effect claims. Failing this, a series of 
carefully considered, related, non-experimental studies including other 
plausible variables need to be conducted to develop stronger claims of 
causality (Abelson, 1995; Asher, 1976). Thus, the potential of secondary 
data analyses lies in the identification of explanatory factors that, with 
further experimental or quasi-experimental research, cause variation in 
student achievement. 

2. Are there clearly established links between large-scale testing agencies and 
researchers who can do secondary data analyses? 

The potential of secondary data analyses will be greatest when close and 
clear links exist between large-scale testing agencies and secondary 
analysis teams (Burkhardt & Schoenfeld, 2003; Pascarella & Terenzini, 
2005). These links should be established before the design of the testing 
program and they should provide for a direct route between policy 
makers, testing specialists responsible for designing and implementing 
an assessment program, and secondary analysis teams. Through these 
links, policy issues that could be addressed in the secondary data 
analyses could be identified prior to data collection. At that time, 
variables relevant to policy could be identified. With this procedure, 
other problems, identified below, could be mitigated. 

3. Who should be the secondary researchers? 

As advocated above, secondary data analysis research involves both 
policy/conceptual and data analysis issues, and should be directed 
toward improving student learning. Thus, to be most relevant, 
secondary analysis research teams should be multidisciplinary teams 
that collectively possess knowledge and skill in each of the following 
components: 

a. policy formulation and implementation, 

b. curriculum. 
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c. instructional procedures and practices, 

d. item development and test and quest! ormaire construction, 

e. administration of tests and survey questionnaires, 

f. establishing performance sfandards and seffing cuf-scores, 

g. classical fesf and ifem response models, 

h. sample designs, sampling, and large-scale sfafisfical analysis 
procedures fhaf will likely need fo incorporafe sampling weighfs and 
knowledge and skill wifh fhe compufer programs needed fo conducf 
fhese analyses; and 

i. wrifing reporfs and arficles for differenf audiences. 

Figure 1 represenfs a modification of fhe deliberative research 
franslafion model proposed by Willinsky (2001). This model shows fhe 
place secondary dafa analysis feams should occupy in a 
research <-> practice sysfem. Inclusion of secondary analysis feams in fhis 
model should enhance sfudenf learning by focusing on fhe needs of 
policy makers and professional developmenf officers, and on improving 
classroom practice. 

Barriers to collaboration. Burkhardf and Schoenfeld (2003) idenfified 
fhree barriers fhaf, if nof addressed, inhibif fhe creafion of an 
infrasfrucfure fhaf will lead fo realistic collaboration in educafional 
research and developmenf. The fhree barriers are: 

a. significanf loss of aufonomy, organizational complexifies, and fhe 
concomifanf expendifure of fime and energy fo esfablish and 
mainfain collaboration, 

b. possible loss of individual sfafus in aufhorship, and 

c. absence of consisfenf funding fo supporf large R & D feams. (p. 11) 

For fhe firsf fwo barriers, Burkhardf and Schoenfeld (2003) suggesf fhaf 
feam members adopf fhe value sysfems of fhe nafural sciences. These 
value sysfems allow feam members fo receive appropriafe credif (e.g., 
joinf recipienf of granfs/confracfs and joinf aufhorship). Wifhouf 
adequafe and susfained funding of secondary analysis feams fhrough 
insfifufional supporf and availability of exfernal funds, fhe pofenfial 
impacf of fhese analyses will nof be realized. 
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Deliberative Research Translation 



Figure 1. Deliberative research translation for secondary data analysis. 


4. How representative are the samples of students, teachers, and schools? 

The sample design used by CMEC for selecting students is a three-stage, 
stratified, unequal probability design. CMEC intends the sample to be a 
representative sample of students, teachers, and schools for each 
participating province and for Canada. Elowever, the actual sample is 
likely not representative because of missing tests and questionnaires and 
non-response at the item level. 

Missing tests and questionnaires. The number of students in the 
database for 13-year-olds for the 2003 CMEC Writing Assessment is 
12,708. Elowever, not all students have school and/or teacher data. 
Deletion of the teachers and schools with missing identification codes, 
which were to be entered by the students, leaves a sample of 11,570 
students. 2 Moreover, an additional 446 students do not have writing 
scores. Thus, the total number of 13-year-old students with teacher and 
school information and writing scores is 11,124 (87.5% of the initial 
sample). Of the 10,972 students assessed in the population of 16-year- 
olds, 8,663 (80.0%) have teacher and school information and writing 
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scores. What is not clear is if the students who remain adequately 
represent the two populations. 

Non-response at the item level. The data file confained a sef of derived 
scores for school, feacher, and sfudenf quesfionnaires. These scores were 
defermined for fhe combined 13-year-old and 16-year-old sample using a 
principal componenfs exfracfion followed by a varimax rofafion of fhe 
componenfs wifh eigenvalues greafer fhan or equal fo one (Kaiser, 1970). 
The number of derived sfudenf variables (weighfed facfor scores) was 
eighf. Only 970 13-year-old sfudenfs had complefe dafa on fhese derived 
variables. Furfher, only 388 ouf of fhe 1,289 feachers of fhese sfudenfs 
had complefe dafa on fhe 30 derived feacher variables. Clearly, fhe 
amounf of missing dafa af fhe ifem level is a concern and raises a more 
serious quesfion abouf fhe represenfafion of fhe sfudenf, feacher, and 
school populafions. 

5. What is the integrity of the results? 

Dafa musf have infegrify so fhaf fhe analysis yields resulfs fhaf can be 
validly inferprefed. Incorrecfly complefing an ifem and ifem non- 
response seriously compromise fhe infegrify of fhe resulfs. In fhe Wrifing 
Assessment for example, sfudenfs recorded fheir gender fwice, once on 
fhe cover page and again in response fo ifem 5 in fhe sfudenf 
quesfiormaire. Shown in Table 1 is a summary of fhese fwo responses for 
fhe 13- and 16-year old samples for fhe Wrifing Assessmenf. As can be 
seen, 24 per cenf of sfudenfs answered fhe fwo quesfions differenfly 
(2,951 13-year-old sfudenfs and 2,533 16-year-old sfudenfs). Furfher, 
fhere were 154 missing gender responses on fhe cover page and 976 
missing responses for ifem 5, which resulfed in 612 13-year-old sfudenfs 
and 492 16-year-old sfudenfs wifh missing dafa for one or bofh of fhe 
gender ifems. If is nof clear which of fhe fwo responses, if eifher, is fo be 
frusfed. 




Pitfalls and Potential of Secondary Data Analysis 765 


Table 1: 

Gender Distribution : 

■ Cover Page vs. Question 5 


Gender: Question 5 

Gender: 

Cover Sheet 

Male 

Female 

Total 

Male 

4,456 (3,960) 

1,388 

(1,118) 

5,166 

(5,078) 

Female 

1,563(1,415) 

4,689 (3,987) 

5,589 

(5,402) 

Total 

6,019 (5,375) 

6,077 (5,105) 

12,096 

(10,480) 


Note: Numbers for 16-year-old sample in parentheses. 


The second example involves the items used to compute the derived 
score for socio-economic status. The SES index is a linear composite of 
mofher's and/or fafher's educafion (quesfion 51) and occupafion 
(quesfion 52) fhaf sfudenfs reporfed on fhe sfudenf quesfiormaire. The 
number of missing cases for the four ifems ranged from 2,472 fo 3,383 af 
age 13 and from 2,365 fo 2,497 af age 16. Consequenfly, fhe SES index 
was available for only 4,391 (34.6%) 13-year-old sfudents and 6,110 
(55.6%) 16-year-old sfudenfs. There are several possible explanafions for 
fhe non-response rates for fhese ifems. Firsf, some sfudenfs likely did nof 
know fheir parenfs' educafional level and/or occupafion. Second, some 
sfudenfs may have omiffed fhese ifems because fhey found if difficulf fo 
classify fheir parenf(s)' occupafion in fhe 10 occupafional categories. 

These examples raise questions about the integrity of fhe 
quesfiormaire dafa. Lisf wise delefion of cases resulfs in a massive 
reduction in fhe number of cases fhaf, in turn, raises questions about 
what populations the samples represent. 
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6. Are the correlations between performance and the derived background 
variables unexpectedly low? 

Student performance on the Writing assessment is scored using a six- 
point scoring rubric. The zero-order correlations between the writing 
scores and the derived variable scores for 13-year-old sfudenfs ranged 
from -0.165 fo 0.236, wifh fhe majorify less fhan| 0.10 | . This can be seen 

in Table 2: 85.9 per cenf of fhe 13-year-olds and 80.0% of fhe 16-year olds 
received scores in fhree adjacenf score cafegories. This resfricfed scaling 
consf rains variabilify, which in fum influences fhe correlations. 


Table 2 


Distribution of Students by Performance Level 


13-year-old 

Sample 

16-year-old 

Sample 

Performance 

Level 

N 

% 

N 

% 

Below 1 

653 

5.1 

590 

5.4 

Level 1 

1,991 

15.7 

895 

8.2 

Level 2 

5,199 

40.9 

3,012 

27.5 

Level 3 

3,718 

29.3 

4,169 

38.0 

Level 4 

607 

4.8 

1,589 

14.5 

Level 5 

47 

0.4 

261 

2.4 

Missing 

493 

3.9 

456 

4.2 

Total 

12,708 

100.0 

10,972 

100.0 

Mean 


2.15 


2.58 

Standard Dev. 


0.94 


1.10 




Pitfalls and Potential of Secondary Data Analysis 


767 


7. Can more advanced statistical procedures be used with the present CMEC 

sample design? 

Organizations like CMEC would like secondary analysis teams to look at 
deeper, more complex policy relevant questions.^ The structure of 
national assessment data sets, such as the CMEC data sets, is typically 
students nested within classes nested within schools nested within 
provinces. The Elierarchical Linear Model (HEM) (Raudenbush & Byrk, 
2002; Luke, 2004) is a statistical procedure that can separate school effects 
from class effecfs from sfudenf effecfs. If fhese level effecfs are nof 
separafed, fhen sfudenf findings could be incorrecf. Given fhe complex 
inferrelafionships among sfudenf, class, and school variables, HEM is 
superior fo ordinary leasf squares regression procedures. 

HEM requires fhaf sample sizes af each level be relafively large fo 
allow esfimafion of effecfs wifh small sampling error. Alfhough some 
agreemenf exisfs fhaf fhe number of higher level unifs like schools 
should be af leasf 30 (Snijders & Bosker, 1999), liffle agreemenf exisfs on 
fhe required number of individuals or sfudenfs wifhin each unif. Willms 
(1992) suggesfed fhaf estimates for schools wifh fewer fhan 20 sfudenfs 
are nof very sfable. For fhe 16-year-old writing sample, fhere are only 13 
school/class combinations wifh 20 or more sfudenfs. Consequenfly, if is 
nof possible fo generalize fhe resulfs yielded by fhe HEM analyses fo fhe 
full population represenfed by fhe inifial samples of schools and 
sfudenfs. 

Obviously a better sampling procedure is required. For example, fhe 
pofenfial for secondary dafa analyses of CMEC dafa would be enhanced 
if fhe lowesf sampling unif were changed from sfudenf fo class. Thaf is, 
classes wifhin schools should be selecfed and all sfudenfs in fhe selecfed 
classes chosen. Some school co-coordinafors indicafed fhaf if would be 
less disruptive if infacf classes of sfudenfs were assessed (Council of 
Minisfers of Education, 2002b, p. 49). As poinfed ouf by Willms (1992), 
esfimafes of school and class effecfs are needed in addition fo sfudenf 
effecfs (p. 49). However, if likely will be necessary fo use a 2-level HEM 
because many schools have only one class, fhereby preventing fhe 
separation of class and school effecfs. 
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CONCLUSION 

We have discussed what we believe to be major pitfalls that work to 
inhibit the potential of secondary dafa analyses of CMEC assessmenf 
dafa sefs. However, we wanf fo emphasize fhaf wifh appropriafe 
changes, fhe pofenfial of fhe CMEC assessmenf program fo inform, in 
felling and meaningful ways, policy makers and users of fhe CMEC 
resulfs can be realized. To effecf fhese changes, we believe fhaf a closer 
link needs fo occur befween CMEC and secondary analysis researchers. 
These researchers need fo work wifh fhe policymakers, fesfing 
specialisfs, and analysis of fhe large-scale fesfing agency during fhe 
design and planning phase. They are in a posifion fo pose significanf 
quesfions and fo idenfify and help ensure fhaf a) fhe variables fo be 
considered are relevanf fo fhe improvemenf of classroom pracfice and 
sfudenf learning; b) assessmenf insfrumenfs, background quesfionnaires, 
and scoring of responses fo fhese insfrumenfs will yield scores fhaf 
reflecf fhe concomifanf variafion expecfed for fhe independenf and 
dependenf variables; and c) fhaf fhe samples are represenfafive of fhe 
populafions fo which fhe resulfs are fo be generalized and of sufficienf 
size fo permif fhe use of fhe complex dafa analysis procedures fhaf yield 
sfable, unbiased resulfs needed fo answer fhe quesfions posed. This 
collaboration should be on-going, beginning wifh fhe formulation of 
quesfions and fhe design of fhese programs, followed by insfrumenf 
developmenf, sample design, and scoring, and fhe analysis phase as 
analyfic models are developed, evaluafed, and refined. We believe fhaf 
fhe following quesfion should shape fhe oufcomes of fhis collaborafion: 
How can the use of secondary data analysis results be increased in schools and 
school districts, thereby leading to improved student learning mediated through 
the informed practice of school leaders and classroom teachers? 

NOTES 

* Anderson, Rogers, Klinger, Ungerlieder, Glickman, and Anderson (2006) 
found similar problems in their analyses of the 2001 CMEC Mathematics 
Assessment. Likewise, Pascarella, and Terenzini (2005), in their analyses of how 
college education affects students, identified similar problems and issues in their 
examination of the secondary data sets they worked with. 
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^ Students who were not currently studying English Language Arts were 
instructed to leave the teacher ID blank, accounting for some of the missing 
identification numbers (Council of Ministers of Education, Canada, 2002a). 

^ The Canada Education Statistics Council (CESC) and the Social Sciences 
and Humanities Research Council (SSHRC) through the Education Research 
Initiative (ERI) clearly identified education, and learning outcomes in particular, 
as high priority areas for research. ERI intended to stimulate empirical research 
and quantitative analysis to link available databases containing student 
assessment information to educational policy issues in Canada. However, CESC 
and SSHRC no longer support ERI. 
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